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Abstract 

The selection of faculty in academic institutions is an 
important process - one that has long-lasting effects on 
an institution’s ability to fulfill its mission. Faculty influence 
the quality of the education delivered, the effectiveness of 
the programs and activities offered, and the financial 
efficiency of the delivery processes. Failed searches waste 
time and incur needless expense. Inadequate searches 
- those that result in candidates who are poorly qualified 
or lack organizational fit can have profound negative impact 
on these three key strategic elements. Hiring the wrong 
person may lead to dysfunctional departments, dissatisfied 
students, and, eventually, repeat efforts. Applying a sound 
process, one that structures the search, identifies and 
relates the selection criteria, allows for qualitative and 
subjective assessments, and encourages full participation 
of search committee members, can enhance the desired 
outcome, i.e., identification of best candidates that will 
contribute to the quality, effectiveness, and efficiency of 
higher education. 

Improving the Faculty Selection Process in 
Higher Education: A Case for the Analytic 
Hierarchy Process 

Selection of faculty in academic institutions is a process 
that has long-lasting effects on an institution’s ability to 
fulfill its mission. Faculty influence the quality of the 
education delivered, the effectiveness of the programs and 
activities offered, and the financial efficiency of the delivery 
processes. Failed searches, when no suitable candidates 
are identified, result in lost time and needless expense. 
Worse are inadequate searches - those that result in 
candidates who are poorly qualified or lack organizational 
fit. In these cases the opportunity for negative impact on 
these three key strategic elements can be substantial. 
Hiring the wrong person frequently results in repeated 
search efforts in the short-term, duplicating the expenses 
of time, money, and effort. 



Cole (1 995) suggested improvements in faculty selection 
result from a “studied approach of the processes” (page 
60). This studied approach must incorporate a collaborative 
assessment of institutional constituents’ needs. This 
approach extends well beyond a piecemeal and subjective 
process that may suffer from personal biases and 
perceptions. The faculty selection process requires a 
structured problem solving methodology that considers 
organizational context. It must constitute a “defensible 
and documentable management accountability system” 
(page 59). Arguing from a total quality management 
perspective. Cole (1995) emphasized the strength of a 
disciplined process analysis, driven by data, and 
continuously improved. Marchese and Lawrence (1987) 
suggested eight activities critical to effective faculty 
searches: (1) rethink the vacancy; (2) establish the 
committee; (3) define the job; (4) conduct the search; (5) 
screen the applicants; (6) interview the candidates; (7) 
consummate the selection; (8) support the selectee. While 
these activities encompass the acf/V/'f/'es of a faculty search, 
it is the processes within these steps that are critical to 
successful recruitment. 

Faculty, those who teach, those who research, and 
those who do both, frequently lack the time to develop 
such a process for selecting their colleagues - a process 
that captures the essence of their definition of a suitable 
colleague. However, as expressed by one frustrated 
appointment seeker, “hiring a new colleague is too important 
to be regarded as an obligation to duck or to get over with 
as quickly as possible” (Gray, 1 999). The selection process 
is two-way; candidates assess institutions as well. A 
shallow, inefficient, or ill-defined selection process can 
make quite an impression (albeit negative). Administrators 
have vested interests to ensure legitimacy and 
documentation of a selection process that can withstand 
potential inspections and challenges from various parties 
and perspectives. 

Hahn (2002) recognized the lack of a structured process 
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for decision making, one based on information, in many 
corporations. Without arguing the pros and cons of the 
business model, higher education institutions are perhaps 
more similar to corporate entities than not in today’s 
environment. To be successful, Hahn argued (from a 
marketing perspective) the need for a structured approach 
to decision making; one that allows necessary trade-offs 
in systematic fashion, with all perspectives and 
considerations included. Having a well-documented 
selection process that facilitates clear articulation of 
important criteria, explicit definition of preferences, 
efficiency in accurately assessing applicant profiles, and 
selection of a well-qualified individual contributes to the 
overall academic well being of an institution. Faculty and 
administrators effectively and efficiently invest their time 
and effort to assist the institution in its drive for sustained 
mission fulfillment. This case review describes such a 
process, incorporating Saaty’s (1990) analytic hierarchy 
process (AHP), a decision methodology applied commonly 
in numerous industries and organizational settings, yet 
having relatively few reports of application in higher 
education. 

Perhaps too frequently, decisions rely on a single 
criterion that serves as the basis for comparison of 
alternatives; an example would be choosing equipment 
based on acquisition cost alone. So long as scales are 
consistent and numeric measures accurately capture 
expected performance, summary statistics may be 
sufficient for decisions involving more than one criterion of 
importance. For example, choosing equipment based on 
acquisition cost, maintenance costs, repair costs, and 
salvage value can be reasonably accomplished by adding 
the costs for each alternative and then choosing the 
equipment having the minimum total cost (so long as the 
time value of money and opportunity costs are 
incorporated). When scales are not consistent, whether 
in direction or unit of measure or magnitude, making 
decisions based on multiple criteria becomes very complex 
and risky. 

Multiple criteria methods, both qualitative and 
quantitative, were developed to better model decision 
scenarios. These vary in their mathematical rigor, validity, 
and design. Simple additive and multiplicative models, 
weighted or not, aggregate scores for each alternative 
across all criteria. The scale inconsistencies mentioned 
above confound these methods, usually requiring forced 
transformation to some arbitrary unit-less scale. 
Difficulties arise when non-ratio scales are included in this 
process; frequently unique ratio scale properties are simply 
assumed for interval, ordinal, and nominal (qualitative) 
data. The resulting summary representative statistics 
may have little validity and hence inappropriate for important 
applications. 

Mollaghasemi and Pet-Edwards (1997) discussed 
multiple criteria methods based on when the decision 



maker’s preference structure becomes crystallized. When 
this structure is articulated beforehand, appropriate 
methods for comparing alternatives across many criteria 
include scoring methods (such as those already 
mentioned), preference-based methods (applying utility 
theory), the analytic hierarchy process, outranking methods 
(e.g., ELECTRE), and goal programming (an extension of 
linear programming). In decision scenarios involving 
progressive articulation of preferences, constraint-based 
approaches prevail (such as the STEP method or interactive 
goal programming). Finally, posterior articulation of 
preferences may require methods such as data 
envelopment analysis. 

The intent of this paper is to show how applying a 
model that is not overly complex and that does legitimately 
aggregate across scales and addresses consistency in 
judgments from multiple participants can serve to formalize 
a decision process, reduce time commitments, create a 
process orientation, document the strategy, and result in 
better decisions. As discussed below, the analytic 
hierarchy process was chosen for these, and more formal 
mathematical, reasons. 

The Analytic Hierarchy Process 

Saaty (1 994) described the analytic hierarchy process 
(AHP) as a decision making approach based on the 
“innate human ability to make sound judgments about 
small problems” (page 21). Desirable characteristics of 
such an approach include simplicity, usefulness for both 
individuals and groups, accommodative of intuition, 
compromise, and consensus building, and without 
prejudice toward specialized skills or knowledge. Saaty 
suggested AHP as a process that requires structuring the 
decision problem to demonstrate key elements and 
relationships that elicits judgments reflecting feelings or 
emotions, and whose judgments can be represented by 
meaningful numbers having ratio properties. These 
numerical representatives can be used to generate weights 
or priorities that represent the relative importance of decision 
criteria. Finally, alternatives can be compared to some 
absolute standard (as was done in this case) or to each 
other such that the comparison results and the criteria 
priorities can be synthesized into single statistics, each 
representing an alternative that can be further analyzed for 
sensitivity to changes in judgments. 

The structure of AHP consists of a hierarchy of criteria 
and sub-criteria cascading from the decision objective or 
goal. By making pairwise comparisons at each level of the 
hierarchy, participants can develop relative weights, called 
priorities, to differentiate the importance of the criteria. The 
scale recommended by Saaty (1994) is 1 through 9, with 
1 meaning no difference in importance of one criterion in 
relation to the other and 9 meaning one criterion is 
extremely more important than the other, with increasing 
degrees of importance in between. Only half the 
comparisons need be made; the “reverse” comparisons 
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simply use the reciprocal values in the matrix of 
comparisons that results. The essence of the AHP 
calculations involves solving an eigenvalue problem involving 
this reciprocal matrix of comparisons. Expert Choice (1 995) 
software performs this task (latest basic version costs 
approximately $600 for higher education); it is also relatively 
simple to approximate the solution in spreadsheet software, 
involving only computations of normalized values and 
averages so long as certain conditions are met (Liberatore 
& Nydick, 1997). The appendix contains Excel 
spreadsheets that demonstrate these calculations for both 
the arithmetic means and geometric means approaches. 

Several issues in the AHP process deserve special 
attention. The first is the consistency of the judgments or 
comparisons. As Saaty (1994) described, the method 
involves redundant comparisons to improve validity 
recognizing that participants may be uncertain or make 
poor judgments in some of the comparisons. This 
redundancy leads to multiple comparisons that may lead 
to numerical inconsistencies. For example, if criterion A 
is just as important as criterion B, then the pairwise 
judgments for A and B to any other criterion should be 
identical. When this doesn’t happen in the judgment 
process, inconsistency can arise. Saaty suggested the 
error in these measurements is tolerable only when it is 
of a lower order of magnitude (10%) than the actual 
measurement itself. Consistency ratios (CR) can be 
calculated and compared to indexes derived from random 
judgments. As long as the CR <= 0.10, analysis can 
proceed. Saaty also emphasized that greater consistency 
does not imply greater accuracy, and judgments should 
be altered only if compatible with one’s understanding. 
Otherwise, more information may be necessary or the 
hierarchy may need reexamination. The Excel spreadsheets 
in the appendix demonstrate sample consistency ratio 
calculations. 

Another potential difficulty is rank reversal, i.e., 
reordering of alternatives when new alternatives, even if 
irrelevant, are introduced. Early in the development of AHP 
this issue was debated in the literature (see, for example, 
Harker & Vargas (1987) and Dyer (1990)). Saaty (1994) 
resolved this issue by defining three different modes of 
AHP, the distributive and ideal modes in the relative 
measurement (pairwise comparison) approach and an 
absolute measurement approach. The process presented 
here used the distributive mode of the relative measurement 
approach in weighting the criteria (which were deemed 
comprehensive and exhaustive), and the absolute 
measurement mode for rating the applicants. Rank reversal 
was not an issue. 

While the focus of this paper is the decision process, 
some further elaboration of these modes will facilitate 
understanding the AHP methodology. In the distributive 
mode, the priorities for the sub-criteria (called local priorities) 
of a given parent criterion at any level of the hierarchy sum 



to one. “It is used when there is dependence among the 
alternatives and a unit priority is distributed among them” 
(Saaty, 1994, p. 130). “The ideal mode is used to obtain 
the single best alternative regardless of what other 
alternatives there are (p. 130).” In this mode, for each 
criterion at each level, the local priorities are divided by the 
largest among them, resulting in one alternative becoming 
ideal wWh a priority of one. Saaty (1 999, p. 130) offers the 
following: “To choose the relevant mode one asks: Do you 
want to choose an alternative that is better relative to the 
others (distributive) or do you want the best of the 
alternatives (ideal).” When interested in the degree of 
difference among alternatives, such as for proportional 
allocation of some benefit, the distributive mode is 
appropriate. To choose just one from many, the ideal 
mode applies. Because the committee members were 
able to develop relevant scales having appropriate units of 
measure for the lowest level criteria, assessment of the 
candidates at this level was achieved by evaluation relative 
to these scales rather than to each other. Hence, the 
absolute mode was the choice. 

Literature Review 

Reports of the AHP methodology, its discussion, and 
its applications number more than 1500 (Lombardo, 2001). 
The pofenf/a/ applications of AHP in higher education are 
numerous and include funding research support requests, 
deciding on sabbatical proposals, assessing performance 
and allocating rewards or compensation, choosing students 
for admission, financial aid, scholarships and awards, and 
evaluating candidates for campus interviews (Liberatore & 
Nydick, 1997). Successful administrative applications have 
included faculty evaluation (Tummala and Sanchez, 1988), 
university strategic planning (Saaty and Rogers, 1976), 
university budgeting (Arbel, 1983), and MBA curriculum 
design (Hope and Sharpe, 1989). Review of the literature, 
however, reveals only a few applications involving personnel 
choices or similar higher education contexts. These include 
a study to differentiate the importance of instructional 
responsibilities, intellectual contributions, and service in 
evaluating business faculty (Ehie & Karathanos, 1994) 
and evaluating potential doctoral programs to identify the 
most appropriate, as a function of the style of institution 
(Tadisina & Bhasin, 1989). 

In a healthcare educational setting, Hemaida and Kalb 
(2001) demonstrated the value of AHP in selecting first- 
year participants in a family practice residency program. 
The decision context required both objective and subjective 
factors; the participation of individuals having managerial 
responsibility fostered reduction of complexity and 
agreement on the results. Another education application 
involved assessment of adult learning preferences (Lee, 
McCool, and Napieralski, 2000). Student participants made 
the requisite pairwise comparisons inherent in the AHP 
methodology, providing the data for the researchers to 
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determine learning preferences among lectures, discussion, 
group projects, and individual projects. Other interesting 
applications in the higher education context include facility 
planning (Benjamin, Ehie, & Omurtag, 1 992), and graduate 
school admissions (Saaty, France, & Valentine, 1991). 

The literature contains other interesting reports of AHP 
applications as well, and while not in a higher education 
setting, these are relevant to the group dynamics frequently 
at play regardless of organizational setting. For example, 
Davies (1994) applied AFIP as a client support aid for 
advertising agency selection. This application incorporated 
perceived levels of individual power and how these 
perceptions influenced overall group preferences. Chan 
and Lynn (1991) studied uses of the AFIP to alleviate 
subjectivity in performance evaluation systems by inviting 
participation from groups having differing perspectives. 
Canada, et al. (1985) presented the use of AHP in making 
career choices; Millet (1998) looked at resolving ethical 
dilemmas that frequently involved divisive and emotional 
issues; and Ross and Nydick (1992) addressed the 
selection of licensing candidates in the pharmaceutical 
industry (Ross & Nydick, 1992). 

The applications of AHP are numerous and varied, with 
consistent reports of success in reducing complex decision 
contexts and incorporating qualitative and subjective criteria 
and assessments. Those related to higher education or 
personnel selection, and in particular faculty selection, 
are indeed few. This case study seeks to mediate this 
paucity by emphasizing the long-term benefits of applying 
AHP in faculty selection; namely, the creation, 
documentation, and application of an efficient and effective 
process that stimulates participation and that results in 
suitable choices acceptable to interested stakeholders. 

Case Study: Improving the Faculty 
Selection Process 

The organizational context for this case was a mid- 
sized, state-related school located in a semi-rural area of 
the northeastern United States. The faculty position involved 
quantitative methods/operations management, set in a 
management department in a college of business. This 
information is important because it helps orient the reader 
to the decision context. First, the position itself was a 
hybrid requiring candidates to have rather broad academic 
preparation and business experience. Second, the 
department had only one of its nine members qualified in 
these fields, the others being primarily organizational 
behavior, human resource management, business ethics, 
and strategy. These mostly qualitative fields differ 
significantly in focus and preparation from the mostly 
quantitative fields in the position at hand. These two 
issues would contribute to the complexity of the decision, 
the conflicting perspectives on requisite skills, and the 
process through which a search and selection would be 
conducted. 



Background 

Detailed procedures and guidelines for faculty searches 
already existed at this university. Affirmative action and 
bargaining unit issues made compliance mandatory, and 
a step-by-step sequence of recruitment activities was well 
defined. Figure 1 illustrates these steps. Other than the 
requirement for a rating sheet that included the criteria 
(and their relative weights, if appropriate) to be applied in 
assessing candidates in the initial review, there was no 
direction concerning how to elicit these criteria, how to 
differentiate their importance, and how to perform the 
actual assessments. This procedure simply listed a 
sequence of activities; it did not convey a process for 
identifying the best candidate from among many applicants. 
For this, faculty committees are left to their own 
experiences, past practices, personal preferences, or any 
other set of events that may emerge from the committee. 
Consistency may be non-existent as search committees 
are frequently ad hoc in their nature, established on a 
vacancy-by-vacancy basis. 



Figure 1 

University-Wide Seiection Process 




While avoiding detailed process definition may seem 
consistent with academic freedom, it is inconsistent with 
the requirements of a quality process discussed and 
mandated in the introduction. Recognizing that the absence 
of any clearly defined assessment process leads to undue 
redundancy, inefficient use of time, needless deliberations, 
and insufficient audit trails, this committee discussed and 
developed the process indicated in Figure 2. It applies to 

Figure 2 

Committee-Defined Seiection Process 




4 



IR Applications, Number 6, Improving the Faculty Selection. . . . 



Figure 3: Committee Preference Structure 
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the latter steps of the process outlined in Figure 1 ; namely, 
“select candidates for further consideration” and “discuss/ 
recommend candidate.” 

Formulate the Preference Structure 

Step 1 . Formulating the preference structure means 
deciding on the selection criteria most important to the 
department of management for the position at hand. 
Committee members should take into account all 
constituents, rather than focus exclusively on their own 
perceptions related to “best fit” definition of a colleague. 
This first step consumes the most time; however, it is the 
most important step. Misspecification of the key criteria or 
factors that are critical to identifying not only suitable, but 
best candidates invalidates all succeeding activities. It is 
basic, foundational, and seminal. The highest-level criteria 
agreed upon by the committee members were (1) 
experience, (2) scholarly activities, (3) technological skills, 
(4) flexibility in teaching capabilities, and (5) experience 
working with diverse populations. 

Next the committee identified sub-criteria (level 2 of the 
hierarchy) for the highest-level criteria (as necessary). For 
example, experience was categorized as education (i.e., 
academic), teaching skills, and business. Teaching skills 
at this level mean quality of classroom instruction (not to 
be confused with the higher level criterion “flexibility in 
teaching capabilities,” which addresses the variety of 
courses for which a candidate is academically and/or 
professionally qualified). Scholarly activities were distilled 
into research record, research potential, and collaboration. 
Technological skills consisted of general purpose and 
application software, and Web-enhanced and online course 
delivery. The remaining two first-level criteria, flexibility in 
teaching capabilities and experience working with diverse 
populations, had no sub-criteria. 

Finally, a third level of sub-criteria was generated for five 
of the level 2 sub-criteria; however, these were identical 
and based on the hybrid nature of the position being 
recruited. The third level sub-criteria for experience, 
quantitative methods (QM) - business statistics (BS)- 
operations management (OM), were common to the three 
second level sub-criteria (education, teaching skills, 
business) and applied to two of the scholarly activity sub- 
criteria (research record and research potential) as well. 
Creating a parsimonious model is always a concern in 



decision analysis and avoiding needless additional sets of 
criteria is appropriate so long as those chosen for application 
across a level of the hierarchy adequately and accurately 
capture key elements of the decision structure. Note that 
the scales associated with these common sub-criteria 
were not, however, the same (discussed later). Figure 3 
depicts the complete hierarchy. 

Differentiate the Importance of the Criteria 

Step 2. The next step in the process is differentiating 
the relative importance of the criteria by completing pairwise 
comparisons for each set of criteria and sub-criteria. Each 
committee member completed software-generated 
questionnaires similar to those depicted in Figure 4. The 
questionnaire at each level of the hierarchy grouped sub- 
criteria for comparison within criteria one level above. For 
example. Figure 4 lists each possible pairwise comparison 
for the five level 1 criteria; a separate questionnaire listed 
the three sub-criteria within the experience criterion, yet 
another listed pairwise comparisons for the three sub- 
criteria within the education sub-criterion, etc. Expert 
Choice (1995), the software that facilitates application of 
AHP, generated these questionnaires; these facilitate 
direct comparison using the 1-9 scale recommended by 



Figure 4 

Sample Pairwise Comparison Questionnaire 
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Figure 5 

Committee Preference Structure with Priorities 
(Levei 1 criteria sum to 1 ; Levei 2 sub-criteria sum to 1 within each Levei 1 criterion, etc.) 
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Saaty (1 994), with numbers to the left of center indicating 
a preference for the criterion listed on the left; the opposite 
being true for the scale on the right. The first column on 
the left simply indexes the comparisons for easy reference 
and cumulative total. Aggregating individual responses via 
consensus or calculating the geometric mean (the nth 
root of the product of the n judgments) (Davies, 1994) 
resulted in the criteria weights, or priorities as they are 
called in AHP, shown in Figure 5. Note that the weights 
for criteria at each level, within their parent (or higher level) 
criterion, sum to one (called local priorities). For example. 
Experience (0.450) was considered approximately twice 
as important as Scholarly Activities (0.243), three times 
more important than Technological Skills (0.145), etc. 
This step prioritizes the criteria in a series of cascading 
allocations within levels and within criteria. Experience 
has sub-criteria Education, Teaching Skills, and Business; 
the priorities at this level were allocated 0.230, 0.648, and 
0.122, respectively, based on the members’ aggregated 
judgments. 

Set Discordance Levels 

Step 3. Setting discordance levels (Mollaghasemi & 
Pet-Edwards, 1997) means establishing minimum 
acceptable scores (representing experience or exposure 
or performance, etc.) for each criterion. For example, the 
committee may have agreed that at least three years of 
university-level teaching experience in business statistics 
was a prerequisite to successful candidacy. Any applicant 
not meeting this minimum level would be excluded from 
further consideration regardless of his or her rating on any 
other criterion. Because this can drastically reduce a pool 



of potential candidates, the committee decided to avoid 
setting such levels, instead relying on the position 
announcement and strict application requirements to 
eliminate marginal or insincere candidates from further 
consideration. Using application requirements in lieu of 
setting discordance levels may eliminate a “diamond in 
the rough” from consideration ; however, in terms of efficient 
use of time and effort, the committee decided it was the 
appropriate approach to follow. Being inundated with Web 
software-triggered resumes and vitas (and the resultant 
high number of very marginal applications) reinforced this 
notion. The check sheet shown in Figure 6 represents the 
“discordance” levels for the application process. Upon 
receipt of initial application materials, applicants received 
an acknowledgement letter specifying missing materials 
and required receipt date. Absence of a checkmark in any 
one cell after this date eliminated the candidate from 
further consideration. 

Create Rating Scales 

Step 4. Determining the appropriate scales to measure 
or assess each candidate can be problematic. The first 
difficulty is the nature of the scale itself and whether one 
already exists. In the latter case, careful definition of 
ratings within the scale helps preclude unintended implicit 
weighting. The committee selected a single verbal scale 
covering all criteria; what differed, depending on the 
criterion, was the scale definition. When possible, objective 
(quantitative) measures were applied to differentiate the 
ratings. The rating scale consisted of these options: 
excellent, very good, good, fair, and poor. As indicated in 
Figure 7, the majority of the ratings were derived based on 



Figure 6: Discordance (yes/no) Checksheet 
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Figure 7: Committee Preference Structure with Priorities and Scaies 
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0.167 


BS 

0.167 


OM 

0.667 


QM 

0.167 


BS 

0.167 


OM 

0.667 


QM 

0.143 


BS 

0.143 


OM 

0.714 


QM 

0.143 


BS 

0.143 


OM 

0.714 








Scale 


# Graduate Courses 


Student Evaluations 
& Peer Reports 


# Months 
Experience 


# Articles, 
Presentations, 
Proceedings 


Narrative Description 


Frequency 


# & Types 


# & Extent 


# Distinct 
Courses 


Cover 
Letter & 
Resume 



frequencies or time intervais. For the experience-education 
criterion, the number of courses studied at the graduate 
levei differentiated the ratings. For experience-business, 
the number of months actualiy practicing a business, 
ranging from none to 24 months, determined the rating. 
The committee attempted to quantify the ratings in this 
manner to the greatest extent possibie subject to retention 
of usefui information. 

Not aii criteria couid be measured appiying this 
technique. Teaching skiiis were evaiuated based on student 
evaiuations and peer reports. These vary substantialiy in 
format and interpretation across institutions. Figure 8 iists 
the rating definitions as measure dependent. For this 
experience-teaching skills criterion, committee members 
reviewed pertinent materials individually, attempting to 
accurately interpret the results (considering issues like 
the types and numbers of questions asked, the response 
scales, and whether comparative analyses such as relative 
standings were included) and then discussed them to 



consensus. The technological skills-general purpose and 
application software criterion was rated using a verbal 
delineation of software, while the scholarly activity-research 
potential and experience working with diverse populations 
criteria were assessed qualitatively based on review of 
application materials such as the research plan, cover 
letter, and resume. Refer to Figure 8 to review the 
completed hierarchy representing the committee’s 
complete preference structure for this search process. 

Rate Each Candidate 

Step 5. Committee members individually reviewed each 
accepted application (recall step 3) and completed a 
rating scorecard for each applicant. Given the majority of 
criteria were rated based on objective measures, there 
was little, if any, disagreement on the appropriate ratings 
for these criteria. Minor differences typically involved 
particular course titles (and what they actually meant), 
how far into the past experience should be reviewed, or 



Figure 8: Committee Preferences Structure with Priorities, Scaies, and Measures 



Level 0 


Identify Best Candidate 


Level 1 


Experience 

0.450 


Scholarly Activities 
0.243 


Technological 

Skills 

0.145 


Flexibility in 
Teaching 
Capabilities 
0.109 


Experience 
with Diverse 
Populations 
0.053 


Level 2 


Education 

0.230 


Teaching Skills 
0.648 


Business 

0.122 


Research 

Record 

0.268 


Research Potential 
0.614 


Collaboration 

0.117 


GP & 

Application 

Software 

0.750 


Web& 

Online 

Course 

Delivery 

0.250 






Level 3 


QM 

0.167 


BS 

0.167 


OM 

0.667 


QM 

0.167 


BS 

0.167 


OM 

0.667 


QM 

0.167 


BS 

0.167 


OM 

0.667 


QM 

0.143 


BS 

0.143 


OM 

0.714 


QM 

0.143 


BS 

0.143 


OM 

0.714 












Scale 


# Graduate Courses 


Student Evaluations & 
Peer Reports 


# Months Experience 


# Articles, 
Presentations, 
Proceedings 


Narrative Description 


Frequency 


# & Types 


# & Extent 


# Distinct 
Courses 


Cover Letter 
& Resume 


Excellent 


4 








24 


4 








4 


Specific 

Application 


4 


4 




Very Good 


3 








18 


3 








3 


SAS 

SPSS 


3 


3 




Good 


2 


Measure Dependent 


12 


2 


Qualitative 


2 


Office Suite 

Excel Add- 
ins 


2 


2 


Qualitative 


Fair 


1 








6 


1 








1 


Word 

Excel 

PowerPoint 


1 


1 




Poor 


0 








0 


0 








0 


None 


0 


0 
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clarification of technology-related activities. For the criteria 
rated qualitatively, members discussed individual ratings 
until achieving consensus. There was little disagreement. 
This was attributed to the well-defined position 
announcement that made very clear exactly what the 
position entailed, the minimum qualifications, and the 
exacting nature of the application materials. Nearly all 
accepted applicants presented detailed, descriptive, and 
clear reviews and explanations of their research goals and 
projects. 

By far the most difficult criterion to rate was experience- 
teaching skills. The variety of formats and scales in 
student evaluations led to interesting debates about what 
the results really indicated. Overall, the rating process 
was efficient and the time required was less than expected. 
Time and effort spent developing the hierarchy and eliciting 
the preference structure facilitated the rating process, 
arguably without any loss of effectiveness or accuracy. 

Determine Initial Ranking 

Step 6. This step involved application of the AHP 
algorithm, which was accomplished using Expert Choice 
(1995) software. This software is not mandatory; ratings 
can be developed using spreadsheet software as well, so 
long as the consistency (see below) among comparisons 
is high. Figure 9 shows the results for the eight candidates 
that emerged successfully from the application review 
(step 3) and the rating process (step 5). 

Select Candidates for Further Consideration 

Step 7. At this stage, the committee reviewed the 
results and decided to continue consideration of the top 
five candidates. This step can incorporate direct 
comparison of candidates (pairwise) on each criterion if 
so desired, rather than relying exclusively on the 



Figure 9: Final Ratings 




Candidate 

comparisons to the absolute scales already completed. 
The process is identical to that described in step 2; the 
software-generated questionnaires would be modified to 
include candidate comparisons on each of the lowest 



level criteria. These results would be aggregated within the 
program and relative rankings again assigned. The results, 
in terms of rank order, would be the same so long as the 
earlier rating process was handled diligently. For example, 
two candidates having the same number of graduate 
courses or having the same length of business experience 
should have received identical ratings on those criteria. 
While the summary numeric ratings (and hence the 
difference among candidate scores) may change, especially 
considering the qualitatively assessed criteria, the order 
would not. 

Measuring Inconsistency of Judgments 

Saaty (1994) offered suggestions to maintain 
consistency in judgments: homogeneity of the elements 
(criteria or alternatives) being compared; sparseness of 
elements being compared; and knowledge and care of the 
decision maker. The process implemented in this search 
committee’s quest to identify the best candidate 
incorporated these traits. The criteria are not so different 
as to make comparison illegitimate or inappropriate, nor 
were the backgrounds in education and experience disparate 
among the applicants. The numbers of both criteria and 
candidates were kept reasonably small, the latter being 
achieved by the rigor of the position announcement and 
application process. Finally, each member of the committee 
was a member of the department and definitely had a 
vested interest in the quality and fit of the candidate 
chosen for the position. These factors were reflected in the 
consistency ratios (the relevant AHP statistic) computed 
for the different sets of pairwise comparisons, with a 
highest of 0.07 and the majority below 0.02. All were well 
within the tolerance specified in the AHP literature. 

Results 

Application of the AHP methodology, principally 
constructing hierarchies, establishing priorities, and 
verifying logical consistency (Saaty, 1999) had several 
worthwhile outcomes that can be grouped into three 
categories: decision process, group dynamic, and decision 
outcome. As mentioned at the outset, the existing 
guidelines and procedures for faculty search committees 
prescribed the what to do. They did not provide any 
guidance or procedures on the how. Each time a search 
committee was organized, unless it consisted of the 
exact same faculty having experience working on a very 
similar position, committee members either (a) spent little 
time organizing a process to ensure reasonably accurate 
identification of the best candidate(s) or (b) spent too 
much time performing this function leading to frustration or 
disinterest because of time and effort inefficiencies. 
Applying the sequential steps based on AHP principles 
supported relatively quick and painless organization of the 
tasks at hand. As a result, a clearly documented decision 
process led to efficient focus of committee members’ time 
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and effort in evaluating the qualitative and subjective (i.e., 
difficult and important) aspects of the candidates’ 
qualifications. Finally, the Director of Social Equity, after 
reviewing the file materials expressed, with delight, that 
the process was without a doubt the best she had ever 
seen. 

As indicated in the introduction, faculty members within 
the department of management had substantially different 
education, experience, and research activities, with the 
majority having limited knowledge and experience in the 
field being recruited. Other differences in perspectives 
were evident from past issues in the department, so group 
cohesiveness could not be expected automatically. Time, 
however, is always limiting, and this case was no exception. 
By delineating the process, linking the position 
announcement to the initial application review, and 
proceeding directly to defining the preference structure 
(both criteria and their relative weights), committee members 
very quickly arrived at group cohesiveness in identifying 
and resolving differences of opinion and judgments. 

Finally, the ranked list that resulted from this process 
clearly differentiated the candidates, and the sensitivity 
analysis techniques available in the Expert Choice (1 995) 
software enabled anyone interested in the outcome to 
understand the influence of the criteria selected and the 
priorities derived. Some faculty leaned much more heavily 
toward teaching experience; others, toward research record 
or potential. Some felt the ability to teach operations 
management was most important; others, business 
statistics. By reviewing the committee’s work, detailed in 
various outputs and notes, it became clear to department 
colleagues that the committee had executed its 
responsibility in a legitimate fashion that incorporated all 
interests. 

Conclusions 

Chan and Lynn (1991) reported the improved 
organizational climate resulting from participants 
internalizing organizational goals by participating in 
establishing the preference structure represented by the 
AHP hierarchy and priorities and the perception of fairness 
and equity that was absent when more arbitrary methods 
of assessing multiple criteria were applied. Djang (1 993) 
discussed the value of decomposing complex decisions 
using AHP and the ease of assessment using structured 
interviews or questionnaires (reference Figure 4). Liberatore, 
Nydick, and Sanchez (1992) suggested the structured 
approach of AHP could better capture the subjective 
judgments of participants. Millet (1998) found that the 
process could enhance shared understanding of decisions, 
save time, and facilitate consensus. Liberatore and Nydick 
(1 997) noted that AHP forced individuals to think through 
the strength of relationships leading to a process that was 
less biased and less political, and had more consistency 
than in the past. 



Saaty (1989) argued that AHP facilitates honest 
exploration of a decision by expression of judgments 
according to some preference intensity that leads to a 
choice that rigorously captures these intensities in 
comparative magnitudes. Permitting tolerable 
inconsistencies in judgments is not limiting; rather, it is 
reality. The AHP is a method that treats group decision 
making and individual decision making consistently. In 
their psychometric comparison of decompositional 
approaches (such as AHP) and holistic approaches, Morera 
and Budescu (1 998) found the former better support defining 
the decision context, allow for considering a larger number 
of attributes, can be attacked and defended, and permit 
more in-depth sensitivity analysis. Saaty (1 994) supported 
this preference for decomposition, adding the importance 
of the synthesis phase, and noting how many models fail 
to execute it in ways that maintain the advantage of 
comparative ratio data. 

Experience reported here agrees with these benefits. 
What should be viewed as a critical, positive, and worthwhile 
process, i.e., faculty recruitment, can become a tedious, 
time consuming, and frustrating one for all parties involved 
(committee members, administrators, and applicants) when 
it is not well-defined, effective, and efficient. It needs to 
result in qualified best candidates. It needs to minimize 
consumption of resources. It needs to capture all pertinent 
preference issues. It needs to be fair and equitable to all 
participants (faculty and applicants). And finally, it needs 
to be reusable (which is where real efficiency manifests 
itself). The process articulated here satisfies these 
objectives. 

Application or imitation of this process demands 
recognition of its limitations as well. “Selling” the 
methodology in unfriendly or unfamiliar environments can 
be difficult and may be perceived as oversimplification (by 
quantification) or time consuming. Technical issues such 
as the type of measurement (relative or absolute) and the 
mode (distributive or ideal) may require supplemental 
investigation by at least one member of any decision 
committee to ensure proper application of methodology. 
Using application software clearly simplifies the 
methodology and moderates these issues. Rank reversal, 
if it is an issue, must be addressed; however, given the 
rather rigid timing, sequence, and review process 
recommended here, the chance for last-minute relevant 
alternatives influencing results is minimal. 

Editor’s Notes 

In the preceding article Grandzol took a technique that 
has achieved a high level of successful use in Operations 
Research and Decision Science, and applied it to a 
problem that we are very likely to face in higher education, 
improving faculty selection. Without his contribution many 
of us would simply have used algebraic averages and 
qualitative consensus building techniques. 
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As he mentioned, this methodoiogy is extremely 
valuable when it comes to setting priorities and selecting 
alternatives in areas such as planning and other 
management activities. In addition to his good work I also 
recommend to you the article from Research in Higher 
Education that he referenced for Liberatore and Nydick 
(1997) where they include further discussion of this 
technique and alternative techniques such as Multitrait 
Utility Theory (MAUT) and goal programming. There is 
also an interesting book on this topic called Decision By 
Objectives (How to convince others that you are right) by 
Foreman and Selly (2001, World Scientific). In addition, 
the use of the terms - “Analytical Hierarchy Process” - 
gives about 13,600 hits with Google. 

There are several points worthy of note. The first point 
is that Analytical Hierarchy Process (AHP) is one of a 
large number of decision choice models that combine a 
hierarchical set of criteria in an analytic manner. As one 
might expect there are also a large number of differing 
variations on each of the primary methodologies. For 
example one can estimate the weights with either an 
arithmetic mean or a geometric mean. The primary concern 
however, is for the institutional researcher to go from 
something such as a set of goals to a set of criteria to a 
set of operational alternatives and to combine these different 
aspects of the decision process into some methodology 
which allows further discussion of priorities and relative 
preference. The AHP seems to be a fairly straightforward 
way for executing this combination. One of the key points 
is that the methodology produces ratio scales from 
preference judgments for the higher-level aspects being 
rated. This is central if we are to conclude that one 
criterion is twice as important as another criterion, or one 
goal is three times as important as another goal. Without 
this ratio scale on higher-level attributes, it makes no 
sense to use metrics from these attributes for weighting 
alternatives. 

Another aspect of AHP is that, as with all pair- 
comparison techniques, it has redundant data. As Grandzol 
mentions, this allows for a check on the consistency of 
the data. While standard software does not seem to have 
an easy method for computing the traditional measure of 
consistency (the author does include an example in his 
appendix), it is likely that techniques such as 
multidimensional scaling can give a close approximation. 
This comes because scaling techniques such as AHP 
are based on the assumption of a single underlying 
dimension. If one wants to use multidimensional scaling 
or principal components from a standard package such 
as SPSS, one rule might be to look at the size of the first 
extracted dimension compared to the remaining 
dimensions. 

While the consistency index deals with the internal 
consistency of a set of ratings, there is also the issue of 
consistency across raters. One way to handle this (as 



Grandzol did) is to consider setting minimum standards 
for the faculty. Another way that he dealt with this challenge 
was to have extensive training sessions and discussions 
with his raters. If someone so chose, the/she could also 
use various nonparametric techniques such as Freedman’s 
Two Way Analysis of Variance for Ranks and Kendall’s 
Coefficient of Concordance to establish the agreement of 
the raters across the objects being rated. The choice here 
is a bit dependent on the participants in the process and 
the audience of reviewers. Keeping the process relatively 
non-mathematical may be preferred. 

There is also an issue of the type of assumptions made 
about the errors in the included ratings and preference 
statements. With a multiplicative model, it is not a safe 
assumption that random errors will cancel out as the 
number of events become larger. One alternative has been 
to assume a log-normal distribution of errors, but the 
addition of the multiplied terms doesn’t go well with this 
assumption. This apparent inconsistency reinforces the 
desirability to find non-parametric methodologies when 
working with these summary decision support tools. 

In conclusion, this article deals with a pressing issue 
of qualitative judgments and it gives us an easy to use 
methodology. It provides an alternative to complex 
judgments by breaking things down into logical components 
and helping us analyze priorities. It also reminds us why 
the decision sciences are a key part of our foundation of 
skills - they add value to the solution of real problems. 
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Appendix: Sample Spreadsheet Calculations for Consistency Indexes 



Level 1 
Level 2 
Level 3 



Criterion 
Criterion 
Criterion 1 
Criterion 2 
Criterion 3 



Experience 

Education 

QM 

BS 

OM 





Education 




Consistency calculations based on the arithmetic mean. 

Average 


Consistency 








QM 


BS 


OM 




QM 


BS 


OM 




(by row) 


Measure 






QM 


1.000 


1.000 


0.250 




0.167 


0.167 


0.167 




0.167 


3.000 






BS 


1.000 


1.000 


0.250 




0.167 


0.167 


0.167 




0.167 


3.000 






OM 


4.000 


4.000 


1.000 




0.667 


0.667 


0.667 




0.667 


3.000 






Education 


This 4.000 indicates that 

education in Operations ^ . normalized 

Management is moderately-to- ., * i /k r \ 

. , X j j X' so they sum to 1 (by column), 

strongly preferred over education ^ ^ ' 

in Quantitative Methods. 

Now introduce inconsistency in comparisons and see result 
in consistency index. 

Average 


0.000 Consistency 

See Saaty (1999, p. 80-84) for 
discussion and random 
consistency table. 

Consistency 






QM 


BS 


OM 




QM 


BS 


OM 




(by row) 


Measure 






QM 


1.000 


3.000 


0.250 




0.188 


0.375 


0.167 




0.243 


3.114 






BS 


0.333 


1.000 


0.250 




0.063 


0.125 


0.167 




0.118 


3.039 






OM 


4.000 


4.000 


1.000 




0.750 


0.500 


0.667 




0.639 


3.261 








Note that the 3.000 in the QM- 
BS cell indicates inconsistency 
as the OM-QM and OM-BS 
comparisons are still identical. 














0.119 Consistency 

The inconsistency in the 
comparisons is reflected in this 
index that now exceeds 
acceptable limits. 


Calculation of Consistency 

1 ) Compute the eigenvalue for the matrix with n rows and select the maximum eigenvalue (V) 

2) The Consistency Index (Cl) is (T\ n)/(n- 1) 

3) The Consistency Ratio (CR) is Cl/RI where Rl is the Random Consistency Index 

For n = 3, Rl = 0.52; n=4, Rl = 0.89; n = 5, Rl = 1 .11 ; n = 6, Rl = 1 .25; n = 7, Rl = 1 .35; n = 8, 


Rl=1.40, n = 


9, Rl=1.45 Saaty (1999) 



Level 1 


Criterion 


Experience 


Level 2 


Criterion 


Education 


Level 3 


Criterion 1 


QM 




Criterion 2 


BS 




Criterion 3 


OM 



Consistency calculations based on the geometric mean using natural logarithms. 



Education 





QM 


BS 


OM 


QM 


1.000 


1.000 


0.250 


BS 


1.000 


1.000 


0.250 


OM 


4.000 


4.000 


1.000 



LN(product)/3 


EXPO 


-0.462 


0.630 


-0.462 


0.630 


0.924 


2.520 



3.780 Sum () 



Average Consistency 

EXP/SUM Measure 



0.167 


3.000 


0.167 


3.000 


0.667 


3.000 



0.000 



Consistency 



This 4.000 indicates that 
education in Operations 
Management is moderately-to- 
strongly preferred over education 
in quantitative Methods. 



These values are derived 
using the natural logarithm 
approach to calculating the 
geometric mean. 



Note the result here is identical to 
that derived using the arithmetic 
mean approach. 



Now introduce inconsistency in comparisons and see result in consistency index. 



Education Average Consistency 





QM 


BS 


OM 


QM 


1.000 


3.000 


0.250 


BS 


0.333 


1.000 


0.250 


OM 


4.000 


4.000 


1.000 



EXP/SUM 


Measure 


0.235 


3.136 


0.113 


3.136 


0.652 


3.136 



LN(product)/3 


EXPO 


-0.096 


0.909 


-0.828 


0.437 


0.924 


2.520 



3.865 Sum () 0.117 Consistency 



Note that the 3.000 in the QM- 
BS cell indicates inconsistency 
as the OM-QM and OM-BS 
comparisons are still identical. 



Here the result is approximately the 
same as that derived using the 
arithmetic mean approach 



Another method to derive geometric means is to multiply the elements by row and take their nth root. The procedure then continues as in the arithmetic mean case. Saaty 
(1994) discusses potential difficulties. 
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